Skip to content

feat(containment): generic out-of-band security webhook and signals contract (#879) - #897

Open
oyesanyf wants to merge 5 commits into
google:mainfrom
oyesanyf:feat/containment-signals-contract
Open

oyesanyf wants to merge 5 commits into
google:mainfrom
oyesanyf:feat/containment-signals-contract

Conversation

@oyesanyf

Copy link
Copy Markdown
Contributor

Resolves #879

Signed-off-by: femi oyesanya oyesanyf@gmail.com


1. Summary & Motivation

Google Agent Mesh operates as the Zero Trust identity, cryptographic task authority, and Policy Enforcement Point (PEP) for multi-agent communications, while delegating OS, pod, and filesystem confinement to external sandbox runtimes (NVIDIA OpenShell, Docker Sandbox docker sbx, Kubernetes agent-sandbox with gvisor/kata, or host supervisors), as documented in site/content/docs/contributing/security-architecture.md §3.1:

"Confinement of the operating system, filesystem, and guest network namespace belongs to the sandbox platform... Agent Mesh provides the cryptographic task authority that those sandboxes consume."

The Gap

When a sandboxed workload is compromised (e.g. via prompt injection or sub-goal hijack) or its identity is revoked:

  1. agentmesh-node blocks unauthorized requests in-band (via Biscuit Datalog policy, TaskAuthorizationRule tar_block narrowing, or ModelArmor/ExtProc inspection) and drops streams from revoked peers (MeshEvent.BANNED).
  2. However, the external sandbox orchestrator or SOC/SIEM has no structured, machine-readable signal that the workload violated security boundaries or was evicted from the mesh — leaving the compromised container/pod running until its TTL expires unless operators scrape unstructured logs.

What This PR Delivers

This PR implements an opt-in, 100% generic, runtime-agnostic out-of-band security webhook contract across agentmesh-node and agentmesh-control-plane:

  • Defines the typed SecuritySignalEvent envelope and four discrete security state transitions in api/agentmesh.proto (protojson with UseProtoNames: true, plus CloudEvents 1.0 binary-mode HTTP headers).
  • Dispatches signals asynchronously via a non-blocking bounded ring buffer with sliding-window burst coalescing over unix:/path/to.sock, https:// (with mTLS / custom CA), or loopback-only http://127.0.0.1:<port>.
  • Adds zero external dependencies (git diff go.mod go.sum is empty) and zero vendor-specific or OS-specific concepts. Every protobuf message introduced in this PR is actively wired end-to-end and covered by unit and integration tests.

2. Addressing Maintainer Feedback on Issue #879

On issue #879, @aojea noted:

"I like the idea of webhooks for integration, but the interface should be generic and I'm still unclear what kind of signals we should use, so a first step should be defining this generic contract"

  1. 100% Generic & Runtime-Agnostic Interface:
    • Core protobuf schemas (api/agentmesh.proto), Go packages (internal/containment), and node/control-plane hooks contain zero references to specific host engines or OS kernel primitives.
    • Standardizes on CNCF CloudEvents 1.0 (Binary Content Mode) headers paired with protojson (UseProtoNames: true) bodies so Knative/Eventarc sinks, Kubernetes controllers, OpenShell/gVisor supervisors, SIEM pipelines, and host daemons can consume events without custom adapters.
  2. Four Discrete, High-Signal Transitions (Never Firing on Allowed Traffic):
    • agentmesh.policy.denied: Biscuit Datalog denial, tar_block narrowing violation, or ModelArmor/ExtProc block, distinguishing LOCAL_WORKLOAD vs REMOTE_PEER.
    • agentmesh.transport.refused: libp2p connection/stream auth gate refusal (unauthenticated, not_router, banned), coalesced over a configurable window to prevent scan floods.
    • agentmesh.identity.revoked: Control-plane revocation or node application of MeshEvent.BANNED / revocation_ids / ban-rejected /refresh.
    • agentmesh.enrollment.rejected: /enroll or /refresh bootstrap/recovery refusal on agentmesh-control-plane.
  3. Focused Scope (100% Wired Code Only):
    • Scoped strictly to outbound asynchronous security signal delivery so inline allow/deny authority remains exclusively in Biscuit Datalog, ext_authz, and Inspection (ExtProc / ModelArmor), and every schema message in api/agentmesh.proto has live emission and test coverage today.

3. Signal Flow

sequenceDiagram
  autonumber
  participant Agent as Untrusted Agent Workload
  participant Node as agentmesh-node PEP
  participant Dispatcher as containment.Dispatcher
  participant Webhook as External Supervisor
  participant Host as Sandbox / Orchestrator Runtime

  Agent->>Node: Unauthorized API / Tool Request
  Note over Node: Evaluate Biscuit Datalog & tar_block
  Node-->>Agent: 403 Forbidden (Immediate Data-Path Rejection)
  
  Note over Node,Dispatcher: Non-Blocking Out-of-Band Dispatch
  Node->>Dispatcher: Dispatch(&api.SecuritySignalEvent{...})
  
  alt Signal Coalescing Window Active
    Dispatcher->>Dispatcher: Aggregate count per (signal, peer, service)
  else Signal Ready for Dispatch
    Dispatcher->>Dispatcher: Enqueue to Bounded Ring Buffer
    Dispatcher->>Dispatcher: Compute HMAC-SHA256 & Format CloudEvents 1.0
    Dispatcher->>Webhook: HTTP POST /events (unix:... or https:...)
    Webhook->>Webhook: Constant-time HMAC verification & ProtoJSON parse
    Webhook->>Host: Enforce Containment (cgroup freeze / Pod evict / Token revoke)
  end
Loading

4. The 4 Outbound Security State Transitions (api/agentmesh.proto)

All signal messages live in api/agentmesh.proto inside the SecuritySignalEvent envelope and obey AGENTS.md §1 (google.protobuf.Timestamp fields named *_time — event_time and first_seen_time; unset instants rejected):

  1. agentmesh.policy.denied (PolicyDeniedSignal):
    • Direction: LOCAL_WORKLOAD vs REMOTE_PEER.
    • Reason (PolicyDenialReason): DATALOG_EVALUATION_DENIED, TASK_AUTHORIZATION_RULE_VIOLATION, INSPECTION_FILTER_BLOCKED, ROLE_MISMATCH, EXPIRED_CREDENTIAL, UNAUTHORIZED_SERVICE, UNAUTHORIZED_PATH_OR_METHOD (classified deterministically via typed errors.Is unwrapping).
    • Context: target_service, requested_operation, requested_resource, caller_peer_id, caller_subject, caller_roles, caller_labels, event_time.
  2. agentmesh.transport.refused (TransportRefusedSignal):
    • Reason (TransportRefusalReason): PEER_UNAUTHENTICATED, NOT_ROUTER_ROLE, AUTH_DEADLINE_EXCEEDED, PEER_BANNED, PROTOCOL_UNAUTHORIZED.
    • Coalescing: Tracks coalesced_count, first_seen_time, and event_time per (signal_type, peer_id, protocol_id).
  3. agentmesh.identity.revoked (IdentityRevokedSignal):
    • Source (RevocationSource): CONTROL_PLANE_ADMIN, MESH_EVENT_BANNED_GOSSIP, LOCAL_CONTROL_PLANE_SYNC, TOKEN_REFRESH_REJECTION.
    • Context: peer_id, reason, revocation_ids, issuer_peer_id, event_time.
  4. agentmesh.enrollment.rejected (EnrollmentRejectedSignal):
    • Reason (EnrollmentRejectionReason): INVALID_JWT, EXPIRED_BOOTSTRAP_TOKEN, BOOTSTRAP_TOKEN_USAGE_EXCEEDED, UNATTESTED_LABELS_REJECTED, INVALID_CHALLENGE_SIGNATURE, REQUESTED_ROLE_UNAUTHORIZED, PEER_PREVIOUSLY_BANNED, AUTONOMOUS_RECOVERY_DISABLED.
    • Context: peer_id, requested_role, bootstrap_token_id, remote_address, submitted_labels, is_autonomous_recovery, event_time.

5. CloudEvents 1.0 Headers, Transport Security, & Secret Hygiene

5.1 CloudEvents 1.0 Binary Mode & HMAC Signing

Each POST carries ce-specversion: 1.0, ce-id, ce-type, ce-source, ce-time, ce-subject, Content-Type: application/json, and (when configured) X-AgentMesh-Timestamp + constant-time X-AgentMesh-Signature: sha256=<hex> over <timestamp>.<body>.

5.2 Transports & Anti-SSRF Enforcement

  • unix:/path/to.sock: Local Unix domain socket transport.
  • https://...: TLS 1.2+ with optional ca_cert_path, client_cert_path, and client_key_path for mTLS.
  • http://127.0.0.1:<port>: Plaintext HTTP is restricted strictly to loopback (net.ParseIP(host).IsLoopback() / localhost); non-loopback http:// targets fail closed at initialization.

5.3 Secret Hygiene (AGENTS.md §1)

  • Secrets are loaded strictly via file paths (--security-webhook-hmac-path, --security-webhook-ca-cert-path, --security-webhook-client-cert-path, --security-webhook-client-key-path) or environment variable (AGENTMESH_CONTAINMENT_HMAC_SECRET, immediately scrubbed via os.Unsetenv after read), never via CLI value flags or echoed in logs.

6. Zero New Dependencies, Observability, & Verification

  • Zero go.mod / go.sum Changes (AGENTS.md §2): git diff go.mod go.sum is 0 bytes.
  • Multi-Language Bindings Synced: api/agentmesh.pb.go, sdk/js/src/gen/agentmesh_pb.ts, and sdk/python/src/agent_mesh/_proto/agentmesh_pb2.py / .pyi.
  • Prometheus Metrics (promauto):
    • agentmesh_containment_events_queued_total
    • agentmesh_containment_events_delivered_total
    • agentmesh_containment_events_dropped_total
    • agentmesh_containment_events_coalesced_total
    • agentmesh_containment_delivery_duration_seconds
    • agentmesh_containment_queue_depth
  • Documentation & Reference Receiver: Hugo reference docs (site/content/docs/reference/containment-signals.md, site/content/docs/preview/sandboxed-agents.md, agentmesh-node.md, node-config.md) and runnable pure-Go receiver (development/examples/generic-containment-webhook/).
  • Idiomatic Go Tests (AGENTS.md §3): Standard testing package only, zero Testify/Ginkgo, zero time.Sleep, -race clean, <10s per suite.
git diff go.mod go.sum
go test -v ./internal/containment/...
go test -v ./development/examples/generic-containment-webhook/...
go test -v ./internal/node -run 'Test(ContainmentSecurity|PolicyDenialClassifier|ContainmentNode)_.*'
go test -v ./internal/controlplane -run 'TestControlPlaneContainment.*'
go build ./cmd/agentmesh-node ./cmd/agentmesh-control-plane ./development/examples/generic-containment-webhook

@oyesanyf
oyesanyf requested a review from aojea as a code owner November 11, 2026 03:07

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a generic security signals and containment framework (Issue #879) allowing Agent Mesh components to dispatch out-of-band security signals to external supervisors via a non-blocking, bounded ring-buffer queue with support for HMAC-SHA256 signing, mTLS, and event coalescing. The review feedback identifies a critical Denial of Service (DoS) vulnerability in the coalescer due to unbounded map and timer allocation from unauthenticated peers, which should be mitigated with a hard limit on active entries. Additionally, several style guide violations were found, including multiple instances of uncanonicalized peer IDs crossing boundaries (violating Section 1) and an ignored error from 'rand.Read' during UUID generation (violating Section 6.3).

Comment thread internal/containment/coalescer.go Resolved
Comment thread internal/node/controlplane_sync.go Resolved
Comment thread internal/containment/coalescer.go Resolved
Comment thread internal/node/node.go Resolved
Comment thread internal/controlplane/server.go Resolved
@oyesanyf

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements an out-of-band security containment signals and webhook mechanism (Issue #879) for Google Agent Mesh, allowing nodes and the control plane to asynchronously dispatch real-time security events (policy denials, transport refusals, identity revocations, and enrollment rejections) to external supervisors. The implementation includes a non-blocking dispatcher with a bounded queue, a sliding-window event coalescer, support for HMAC-SHA256 signing and mTLS, and a runnable reference webhook receiver. Feedback on the changes highlights a bug in both the client and configuration parser where the HMAC secret environment variable fallback is ignored if the secret path is empty, as well as an opportunity to handle oversized payloads more gracefully in the reference webhook receiver by returning a 413 status code.

Comment thread internal/containment/client.go Resolved
Comment thread internal/containment/config.go Resolved
@oyesanyf

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the Generic Security Signals and Containment Contract (Issue #879) to enable out-of-band security containment and supervisor feedback loops. It introduces a non-blocking, asynchronous Dispatcher and Coalescer in the internal/containment package to forward security signals (policy denials, transport refusals, identity revocations, and enrollment rejections) to external supervisors over HTTP, HTTPS, or Unix domain sockets. The signals are formatted as CloudEvents 1.0 compliant ProtoJSON payloads and support HMAC-SHA256 signing and mutual TLS (mTLS) for cryptographic integrity. These signals are integrated into both the agentmesh-node (via stream auth gates, middleware, and egress inspection) and the agentmesh-control-plane (via enrollment and refresh handlers). Extensive unit, integration, concurrency, and resilience tests have been added, alongside comprehensive documentation and a runnable reference webhook receiver. The implementation strictly adheres to the repository's style guides, including proper peer ID canonicalization at boundaries, explicit time units, safe loopback-only defaults for plaintext HTTP, and secret hygiene. I have no feedback to provide as the changes are exceptionally well-designed, thoroughly tested, and fully compliant with all style and architectural guidelines.

…ontract (google#879)

Signed-off-by: femi oyesanya <oyesanyf@gmail.com>
…and review findings

- Clone SecuritySignalEvent on Dispatch, Coalescer.Record, and Client.Send, and capture metric/log fields before channel handoff to eliminate concurrent read/write data race under -race.
- Bound Coalescer active entries to MaxCoalescerEntries (1024) with agentmesh_containment_coalescer_entries gauge and agentmesh_containment_coalescer_overflow_total counter to prevent DoS via spoofed peer IDs.
- Canonicalize peer IDs via peer.Decode across EmitPolicyDenied, emitIdentityRevoked, reconcileBannedPeers, emitEnrollmentRejected, and Coalescer.Record, and check crypto/rand.Read errors.
- Check error returns on Close/Remove, remove trailing colon in scheme validation error string, and remove unreachable exported wrappers to satisfy golangci-lint and deadcode in hack/lint.sh.
- Regenerate sdk/python agentmesh_pb2.py and agentmesh_pb2.pyi matching CI protoc output.

Signed-off-by: femi oyesanya <oyesanyf@gmail.com>
biscuit-go defaults to a 2ms Datalog evaluation timeout, which under
-race on loaded CI runners exceeds the budget and causes spurious
denials in TestIngressDatalogPolicyEvaluation. Match api/datalog_test.go
and api/http_grants_test.go (per f1399f6) by passing an explicit 5s
world duration budget.
…ad limit

- Resolve AGENTMESH_CONTAINMENT_HMAC_SECRET via secrets.FromPathOrEnv in
  containment.FromProto, EndpointFromProto, NewClient, and NewSingleClient
  even when hmac_secret_path is omitted, trimming whitespace and preserving
  fallback across multiple configured endpoints.
- Return 413 Payload Too Large (via io.LimitReader(r.Body, maxBodyBytes+1))
  and defer body close upfront in the generic-containment-webhook example.
- Add TestHMACSecretEnvFallbackWithoutPath unit tests and update oversized
  payload assertions.
…t bound

Add the agentmesh_containment_coalescer_entries gauge and
agentmesh_containment_coalescer_overflow_total counter to the
containment signals reference metrics table, and replace raw LaTeX
delimiters in the coalesced_count description with plain Markdown.

Signed-off-by: femi oyesanya <oyesanyf@gmail.com>
@oyesanyf
oyesanyf force-pushed the feat/containment-signals-contract branch from d88dbb9 to d722997 Compare November 21, 2026 18:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proposal: Out of Band Host Containment and Local Veto Hook for Uncooperative Agents (KillSwitch Integration)

1 participant